Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce O(n) canonicalization algorithm #1670

Draft
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

evanlinjin
Copy link
Member

@evanlinjin evanlinjin commented Nov 3, 2024

Fixes #1665
Replaces #1659

Description

We introduce an O(n) algorithm to determine the canonical set of transactions in TxGraph.

The algorithm traverses graph edges (spends) backwards. We start traversing from transactions with the highest last-seen-in-mempool values as they are most likely to still be in the mempool and not conflict with our confirmed txs.

When a transaction is determined to be canonical, we can also conclude that it's ancestors must be canonical. When a transaction is determined to be non-canonical, we can also conclude that it's descendants are not canonical.

CanonicalIter::canonicalize_by_traversing_backwards is called on txs ordered by LastSeenIn. Anything determined to be canonical/not-canonical are marked in respective sets so we do not duplicate work.

Notes to the reviewers

TODO

Changelog notice

TODO

Checklists

All Submissions:

  • I've signed all my commits
  • I followed the contribution guidelines
  • I ran cargo fmt and cargo clippy before committing

New Features:

  • I've added tests for the new feature
  • I've added docs for the new feature

Bugfixes:

  • This pull request breaks the existing API
  • I've added tests to reproduce the issue which are now passing
  • I'm linking the issue being fixed by this PR

@evanlinjin evanlinjin added the bug Something isn't working label Nov 3, 2024
@evanlinjin evanlinjin added this to the 1.0.0-beta milestone Nov 3, 2024
@evanlinjin evanlinjin self-assigned this Nov 3, 2024
crates/chain/src/tx_graph.rs Outdated Show resolved Hide resolved
crates/chain/src/tx_graph.rs Show resolved Hide resolved
crates/wallet/src/wallet/mod.rs Outdated Show resolved Hide resolved
@notmandatory
Copy link
Member

Per call today @evanlinjin will open a new PR for the 1.0.0-beta milestone that only makes expected breaking changes for Wallet, likely only the ChainPosition enum. Then the algorithm improvements in this PR can be released in a future non-breaking patch version.

@notmandatory notmandatory removed this from the 1.0.0-beta milestone Nov 19, 2024
@notmandatory notmandatory added the audit Suggested as result of external code audit label Nov 19, 2024
@evanlinjin evanlinjin force-pushed the fix/1665 branch 8 times, most recently from 5278b81 to 7610b65 Compare November 19, 2024 06:44
@notmandatory notmandatory added this to the 1.0.0-beta milestone Nov 21, 2024
@notmandatory
Copy link
Member

On further discussion today at release planning call this PR was moved back into the 1.0 milestone if it can be completed and reviewed in time. If not it will have to wait for a 2.0 milestone because it required breaking changes to chain crate APIs that are exposed in the Wallet API.

Alternatively we could remove the following Wallet functions and move any error types from chain into the core module which should insulate the wallet crate from this chain crate API changes.

    /// Get a reference to the inner [`TxGraph`].
    pub fn tx_graph(&self) -> &TxGraph<ConfirmationBlockTime> {
        self.indexed_graph.graph()
    }

    /// Get a reference to the inner [`KeychainTxOutIndex`].
    pub fn spk_index(&self) -> &KeychainTxOutIndex<KeychainKind> {
        &self.indexed_graph.index
    }

    /// Get a reference to the inner [`LocalChain`].
    pub fn local_chain(&self) -> &LocalChain {
        &self.chain
    }

@notmandatory notmandatory added the api A breaking API change label Nov 21, 2024
@evanlinjin
Copy link
Member Author

@notmandatory how does this solution compare with just giving wallet a major version bump?

@notmandatory
Copy link
Member

Removing the above functions and moving required chain error type to core should allow us to do breaking chain crate api changes without having to do a major wallet crate release. I also don't see why Wallet users need to access the inner chain types directly instead of using higher level functions like transactions(), or we can add new helper functions without breaking any APIs.

But all that said if it's less risky to just keep everything as is and do a 2.0 release in 6 mo or so I'd be fine with that too.

Change `ChainPosition` to be able to represent transitive anchors and
unconfirm-without-last-seen values.
Previously, the field `TxGraph::anchors` existed because we assumed
there was use in having a partially-chronological order of transactions
there. Turns out it wasn't used at all.

This commit removes anchors from the `txs` field and reworks `anchors`
field to be a map of `txid -> set<anchors>`.

This is a breaking change since the signature of `all_anchors()` is
changed.

Update `TxGraph` field docs for `empty_outspends` and `empty_anchors`.
This is an O(n) algorithm to determine the canonical set of txids.
Add `run_until_finished` methods for `TxAncestors` and `TxDescendants`.
This is useful for traversing until the internal closure returns `None`.

Signatures of `TxAncestors` and `TxDescendants` are changed to enforce
generic bounds in the type definition.
@evanlinjin evanlinjin changed the title Fixes #1665: Introduce O(n) canonicalization algorithm Nov 26, 2024
`TxGraph::ordered_txs` contains txs ordered by `LastSeenIn` values. To
have this change, `insert_anchor` and `insert_seen_at` methods populate
the `ordered_txs` field. Generic constaints needed to be tightened as
these methods need to be aware of the anchor height to create
`LastSeenIn`.

The order of `LastSeenIn` variants were wrong and this is now fixed.
However, a bug was detected where conflicts with transitively-anchored
txs were not detected. The logic of `CanonicalIter` is changed to look
at txs with anchors first.
Comment on lines +245 to +256
// let anchor = if height == 0 {
// ChainPosition::Unconfirmed { last_seen: Some(0) }
// } else {
// ChainPosition::Confirmed {
// anchor: ConfirmationBlockTime {
// block_id: latest_cp.block_id(),
// confirmation_time: 0,
// },
// transitively: None,
// }
// };
// receive_output(wallet, value, anchor)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

woops.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api A breaking API change audit Suggested as result of external code audit bug Something isn't working module-blockchain
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

Performance issue with get_chain_position and associated methods for unconfirmed transactions
2 participants